Efficient Large Scale Continuous Selection-Join Queries Based on Multidimensional Index

We consider the problem of large number of continuous selection-join queries over data streams. As far as we know, in current data stream management systems, events are filtered based on the query plan(s) which are created according to continuous queries defined by users. Even many kinds of optimizations on query plans have been proposed, there are few proposals on the processing of continuous selection-join queries. The query plan-based processing of continuous selection-join queries schedules the executions of operators and manages the buffers for intermediate results used by selection or join operators. When the number of continuous selection-join queries becomes larger, the processing based on query plan is inefficient for two reasons: 1)the system has to schedule large amount of operators, and 2) the memory will be consumed quickly by the larger number of buffers used by the operators. In this paper, we introduce an efficient event filtering algorithm based on multidimensional index structure, where the event filtering is considered as a query in multidimensional space. Because our proposal is not based on query plan, no operator scheduling is needed. At the same time, only one global buffer for join operation is kept, accordingly much larger sliding window can be defined. We first introduce an event filtering model for continuous selection-join queries. Based on the model, an optimized event filtering algorithm is proposed to filter the stream of join results efficiently. The evaluation results show that the event filtering of large number of continuous selection-join queries based on our model has good scalabilities with respect to the number of the queries and number of dimensions. The performance of the event filtering is improved significantly by the proposed algorithm, especially for the continuous selection-join queries with large sliding window.